Reply to Reshef et al.: Falsifiability or bust.

The term " equitability " was introduced by Reshef et al. in ref. 1 to describe measures of statistical dependence that " give similar scores to equally noisy relationships of different types. " Their paper also introduced a new statistic, the " maximal information coefficient " (MIC), that was said to satisfy this equitability criterion. There has since been much interest in MIC, due primarily to its claimed equitability (2, 3). However, neither the original paper (1) nor follow-up work (4) provided an unambiguous mathematical definition of equitability. In particular, the types of noise permissible in the noisy relationships used to define equitability were not described. A recent paper of ours (5) critically examines the claim of ref. 1 that MIC is equitable. To do this, it was necessary to first pin down a precise mathematical definition of equita-bility. We therefore introduce a criterion, called " R 2-equitability, " that is mathematically rigorous and follows naturally from the description of equitability given in the text and figures of ref. 1. We then prove that R 2-equi-tability cannot be satisfied by any dependence measure, including MIC. We conclude that a definition of equitability different from the one suggested by Reshef et al. is needed. The present letter of Reshef et al. (6) disputes the relevance of R 2-equitability to the claims made in their original paper (1). They do not object to our specific mathematical definition. Rather, Reshef et al. now state that the claimed equitability of MIC was only intended to describe a qualitative tendency that they observed when analyzing some data that they themselves simulated. We find this objection of theirs troubling, as it implies that the central claim of ref. 1—that MIC is equitable—was never meant to be falsifiable. Their letter also suggests that we would " toss out " the heuristic notion of equita-bility. The opposite is true. Our paper explicitly argues that equitability is an important concept in data analysis and deserves a proper formalization. After identifying fundamental problems with the R 2-equitability criterion, we propose replacing it with an alternative mathematical criterion called " self-equitability. " Self-equitabil-ity uses the same definition of noise as R 2-equitability but, unlike R 2-equitability, it is satisfiable. In particular, self-equitability is satisfied by mutual information, a fundamental measure of dependence in information theory. MIC, however, violates self-equitability. Based on these mathematical results, as well as supporting simulations (5), we …